1. Field of the Invention
The present invention relates to a stereo image display apparatus (i.e. three dimensional image display apparatus) which can display a stereo image to the viewer with left eye and right eye images with a binocular parallax and, more particularly, to improvements in the stereo image display apparatus for alleviating the departure from the natural sense of viewing and fatigue of the viewer.
2. Discussion of Related Art
As visual display apparatuses or systems, various stereo image display apparatuses for displaying images viewed as stereo images have been proposed.
FIG. 18 is a perspective view showing a head-mounted display (HMD) 700 as an example of such stereo video display apparatus. The illustrated HMD 700 is a binocular stereo display. The HMD 700 has a frame 702, which is mounted on the viewer""s head and supports left and right display elements and also left and right enlarging optical systems 701 in front of the viewer""s left and right eyes. Thus, a left eye image is displayed to the left eye, while a right eye image is displayed to the right eye, whereby the viewer can view the images as stereo image. The frame 702 has a sensor support 703 supporting a head motion sensor 704, which is located on the head to and detects motion of the head. Thus, the viewer can view the image in correspondence to the motion of his or her head. A data processor 720 is connected via a cable 722 to a connector 706, which is supported on a connector support 705 provided on the frame 702. A loudspeaker 709 for outputting sound is provided around the ear. The data processor 720 has operating buttons 720a which are operable by the user for various operations. With the above stereo image display apparatus such as the HMD, the viewing distance and the vergence distance fail to coincide with each other, resulting in a departure from the natural sense of viewing.
FIGS. 19(a) to 19(c) are views for describing how left eye and right eye images are viewed as a stereo image in the stereo image display apparatus. These figures show an example of a stereo image viewed by the left and right eyes. The image includes two objects, i.e., a sphere and a triangular pyramid, the sphere approaching the viewer. In this case, the left eye image and right eye image are changed from those shown in FIG. 19(a) to those shown in FIG. 19(b) and then to those shown in FIG. 19(c). As shown, the sphere is moved toward the center while being gradually increased in size. This means that the binocular parallax is gradually increased.
FIG. 20 shows the way in which the images shown in FIGS. 19(a) to 19(c) are viewed with the two eyes. Increasing binocular parallax leads to verging (or merging, i.e. reaching or going to each a viewer""s state of perceiving one image on the basis of a plurality of images), so that the viewer""s eyeballs are turned inward. This rotation of the eyes is called vergence and is defined as the vergence angle. In this specification, the distance between the optical axes of the eyeballs in vergence and each eye is called parallax distance. In the HMD, the parallax distance is equal to the distance between the point of intersection of the main beams of the left and right images and the main plane of the eyepiece optical system. The parallax of eyeballs immediately induces accommodation. With increasing parallax angle, the accommodation tends to be closer. Conversely, with reducing parallax angle, the accommodation tends to be further apart. In the stereo image display, the plane in which the image can be viewed with the best contrast is fixed. (In this specification, the distance form this plane to each eyeball is called viewing distance.) In this connection, inconsistency has heretofore taken place. Specifically, the above phenomenon occurs not only in the HMD but also in various stereo television sets, such as those of shutter switching system, lenticular system, etc. In these systems, the viewing distance of stereo television is the distance from the display surface of the display, such as a CRT, to each eye of the viewer.
Viewing image of great vergence distance changes as the stereo image in a state that the viewing distance and the vergence distance do not coincide, leads to a problem that the viewing is unnatural. This problem may be avoided by producing an image with less fly-out changes. By so doing, however, the impact of the image as a stereo image is weakened.
To solve this problem, Japanese Patent Publication Heisei 6-85590 proposes an HMD, in which the viewing distance is varied according to the image motion through mechanical driving of the eyepiece lens. Japanese Laid-Open Patent Publication Heisei 3-292093 discloses a method of varying the viewing degree by detecting a point viewed by the viewer and moving lenses according to the depth information at the viewed point. These systems permit the viewing degree and the vergence angle to be coincident.
Japanese Laid-Open Patent Publication Heisei 7-167633 shows a method of controlling the optimum viewing point, which permits the viewer to perceive the depth world of an object in the broadest range, by calculating the point from the binocular parallax of the image, such that the point is reproduced on the surface of a stereo image display unit or at a specified distance from the surface. As a specific means, a parallax map is calculated from the left and right images by using a correlation matching map, and then the mean parallax of the entire image or a weighted mean of image center data is calculated. Using this mean average, a parallax controller controls the horizontal read timing of the left and right images. This method does not require any mechanical drive system, and it is thus possible to prevent size increase.
FIGS. 21(a) to 21(c) are views showing left eye and right eye images displayed in a stereo image display apparatus, which was proposed earlier by the inventor (Japanese Patent Application Heisei 8-28856). Like the case of FIGS. 19(a) to 19(c), two objects, i.e., a sphere and a triangular pyramid, are displayed, the sphere becoming closer. In this case, the left eye and right eye images are changed from those shown in FIG. 21(a) to those shown in FIG. 21(b) and then to those shown in FIG. 21(c). In this apparatus, the parallax of the left eye and right eye images is substantially fixed irrespective of the motion of the sphere as an object toward and away from the viewer.
FIG. 22 shows the three-dimensional space made by an HMD when the images of FIGS. 21(a) to 21(c) are displayed on the HMD. In this case, the vergence distance L with respect to the sphere is unchanged although the image of the ball is increased as the ball becomes closer. The triangular pyramid, on the other hand, is moved apart from the viewer although its size is unchanged. In other words, the distance difference between the triangular pyramid and the sphere is increased as in the prior art case. Nevertheless, the vergence distance L with respect to the sphere is substantially fixed.
This is owing to the fact that the human""s eyes is not so sensitive with respect to the detection of the absolute distance although they are sensitive to changes in the relative distance. Experiments conducted by the inventor prove that the viewer viewing a stereo image of only a single object with changing binocular parallax (background being black), cannot perceive distance changes. However, the sense of stereo arises when objects in different motions are displayed simultaneously. This means that it is difficult to recognize a distance change of a single object, although distance changes between two objects can be recognized. According to the proposal noted above, with the distance difference between the sphere and the triangular pyramid changing as usual and also the sphere changing in size while the triangular pyramid is not, the viewer perceive as though the sphere is becoming closer while the triangular pyramid is changing the position thereof. Preferably, the vergence distance L of the sphere in FIG. 22 is made coincident with the viewing distance. More preferably, an eye detector judges whether the viewer is viewing the sphere or the triangular pyramid, and the vergence distance of the image being viewed is made substantially constant.
FIG. 23 is a view for explaining the status of merging of a stereo image, which is actually displayed on a left and a right display surface. The relation between the binocular parallax and the vergence distance L when viewing the stereo image is now considered. Whit reference to the figure, when the merging is attained, the horizontal positions X1 and X2 of the sphere on the left and right display surfaces when the sphere is viewed to be at a vergence distance L and on a horizontal position xe2x88x92H, are respectively given as equations (1) and (2).
X1=(d+(xe2x88x92H)/Lxc2x71/tan xcex8)xe2x80x83xe2x80x83(1)
X2=(xe2x88x92d+(xe2x88x92H)/Lxc2x71/tan xcex8)xe2x80x83xe2x80x83(2)
In these equations, d is the distance from the mid point between a left and a right lens to each lens (the distance being positive for the right eye and negative for the left eye), and xcex8 is the half field angle. The horizontal positions are prescribed as follows.
FIG. 24 is a view showing how the horizontal positions X1 and X2 in FIG. 23 are prescribed. As shown in FIG. 24, the prescription is made by setting the horizontal center value of the display region to xe2x80x9c0xe2x80x9d and the horizontal length of the display region to xe2x80x9c2xe2x80x9d. Equation (1) can be derived from the fact that the triangle with points A to C in FIG. 23 as the apices and the triangle with origin 0 and points X1 and C on the left display surfaces as the apices are similar to each other. Likewise, equation (2) can be derived from the similarity of the triangle with points D, B and E as the apices and the triangle with the origin 0 and points X2 and E on the right display surface to each other.
Equations (1) and (2) can be rearranged into equation (3).
X1xe2x88x92X2=(2D/L)xc2x71/tan xcex8xe2x80x83xe2x80x83(3)
In equation (3), the left side (x1xe2x88x92x2) represents the parallax. Equation (3) shows that the vergence distance L when the merging is attained is independent on the horizontal position H but is determined when the parallax is determined.
The permissible change in the vergence distance L, i.e., the permissible change in the parallax, will now be considered. FIG. 25 is a view showing the correspondence relation between accommodation (i.e., state of focus of the eyes) and vergence. The figure shows the permissible range of the vergence-accommodation and the parallax (xe2x80x9cO Plus Exe2x80x9d, Physiological Optic 15, December 1985, pp. 103). In the graph, the ordinate is taken for the accommodation (parallax)) (D: diopter), and the abscissa is taken for the vergence (vergence angle: MW). It will be seen from the graph that the vergence is obtainable in a short period of time so long as its changes are within 4 diopters.
In the prior art techniques or the earlier system proposed by the inventor as described above, the binocular parallax which is based on the left eye and right eye images, is not controlled in direct dependence on the instantaneous accommodation (i.e., state of focus of the viewer""s eyes) or vergence. Therefore, sufficient design for alleviating the departure of the natural sense of viewing or fatigue of the viewer may be relatively difficult.
With the construction of the system shown in the Japanese Laid-Open Patent Publication Heisei 7-167633 which, for permitting the viewer to perceive the depth world of an object in the broadest range, controls the optimum viewing point by calculating the point from the binocular parallax of image, specifically calculating a parallax map from the left and right images by using a correlation matching map and then calculating the mean parallax of the entire image or a weighted mean as the product of image center data and a weight, so that the point would be reproduced on a stereo image display surface or at a specified distance therefrom, such problems are posed as a cost increase due to the necessity of a plurality of frame memories and transmission capacity increase for transmitting left and right image signals and parallax map signal.
In view of the above problems inherent in the prior art, the present invention has an object of providing a stereo image display apparatus of the type as shown above, which permits binocular parallax control sufficiently reflecting the instantaneous accommodation or vergence of the viewer""s eyes and further reduction of the calculation time, cost and data transmission capacity.
According to the present invention, a stereo image display apparatus of the type as shown above can be realized, which permits binocular parallax control sufficiently reflecting the state of the instantaneous accommodation or vergence, as well as permitting further reduction of the calculation time, cost and data transmission capacity.
The constructions according to the present invention, problems which can be solved by these constructions and advantages obtainable according to the present invention, in various aspects contained in the specification, are summarized as follows.
According to a first aspect of the present invention, there is provided a stereo image display apparatus comprising: display means capable of displaying left eye and right eye images with binocular parallax in predetermined display regions; eyeball accommodation detecting means for detecting a viewer""s eyeball accommodation; and binocular parallax control means for effectively controlling the binocular parallax according to the eyeball accommodation detected by the eyeball accommodation detecting means with utilizing some way bringing about a resultant change in the binocular parallax.
In the prior art, the optimum viewing point which permits the viewer to sense the depth world of an object in the broadest range is calculated from the binocular parallax of image, and a control is made such that the optimum viewing point is reproduced on the surface of a stereo image display or at a designated distance from-that surface. However, the calculation of the optimum viewing point is time-consuming, and it has been difficult to make quick parallax control.
The first aspect of the present invention provides a stereo image display apparatus, which is simple in construction and excellent in response. According to the first aspect of the present invention, the viewer""s eyeball accommodation is detected, and the binocular parallax is matched to the detected value, thus permitting realization of a stereo image display apparatus which is simple in construction and excellent in response.
According to a second aspect of the present invention, there is provided the stereo image display apparatus according to the first aspect, wherein the binocular parallax control means controls the binocular parallax by changing the horizontal display positions of the left eye and right eye images in the predetermined display regions.
According to the second aspect of the present invention, in addition to the advantages obtainable according to the first aspect of the present invention, it is possible to realize an apparatus, which is free from mechanical drive, small in size and light in weight.
According to a third aspect of the present invention, there is provided the stereo image display apparatus according to the first aspect, wherein the binocular parallax control means compares a predetermined desired eyeball accommodation and the eyeball accommodation detected by the eyeball accommodation detecting means, controls the binocular parallax to be lower when the detected accommodation corresponds to a point close to the point of the desired accommodation, and controls the binocular parallax to be higher when the detected accommodation is remoter than the desired accommodation.
According to the third aspect of the present invention, in addition to the advantages obtainable according to the first aspect of the present invention, a specific method of the binocular parallax control is provided.
According to a fourth aspect of the present invention, there is provided the stereo image display apparatus according to the first aspect, wherein the binocular parallax control means controls the binocular parallax according to pertinent data held in necessary change data holding means holding data representing various correspondence relations between the result of comparison between the predetermined desired eyeball accommodation and the eyeball accommodation detected by the eyeball accommodation detecting means and a corresponding necessary change in the horizontal distance between the display positions of the left eye and right eye images displayed by the display means.
According to the fourth aspect of the present invention, in addition to the advantages obtainable according to the first aspect of the present invention, adequate binocular parallax control can be quickly made according to pertinent data held in necessary change data holding means such as a ROM.
According to a fifth aspect of the present invention, there is provided the stereo image display apparatus according to the first aspect, wherein the binocular parallax control means controls the binocular parallax to make zero the result of comparison of the predetermined desired eyeball accommodation and the eyeball accommodation detected by the eyeball accommodation detecting means.
According to the fifth aspect of the present invention, in addition to the advantages obtainable according to the first aspect of the present invention, feedback control or like control is made to make zero the difference between a desired eyeball accommodation and an eyeball accommodation detected by eyeball accommodation detecting means as a comparison result, and it is thus possible to obtain adequate binocular parallax control following circumstance changes.
According to a sixth aspect of the present invention, there is provided a stereo image display apparatus comprising: display means capable of displaying left eye and right eye images with binocular parallax in predetermined display regions; eyeball vergence angle detecting means for detecting the viewer""s eyeball vergence angle; desired vergence angle setting means for setting a desired vergence angle concerning images displayed on the display means; and horizontal display control means for controlling the horizontal display positions of a left eye and a right eye image in the predetermined display regions of the display means by shifting the horizontal display positions in opposite directions by an equal absolute amount according to the difference between the desired eyeball vergence angle set by the desired vergence angle setting means and the eyeball vergence angle detected by the eyeball vergence angle setting means.
According to the sixth aspect of the present invention, in the binocular parallax control the horizontal display positions of left eye and right eye images are shifted in opposite directions by an equal absolute amount, so that the horizontal positions of images viewed as stereo image are not undesirably changed.
According to a seventh aspect of the present invention, there is provided the stereo image display apparatus according to the sixth aspect, wherein the binocular parallax control means controls the binocular parallax according to pertinent data held in necessary change data holding means holding data representing various correspondence relations between the result of comparison between the predetermined desired eyeball accommodation and the eyeball accommodation detected by the eyeball accommodation detecting means and a corresponding necessary change in the horizontal distance between the display positions of the left eye and right eye image displayed by the display means.
According to the seventh aspect of the present invention, in addition to the advantages obtainable according to the sixth aspect of the present invention, adequate binocular parallax control can be quickly made according to pertinent data held in necessary change data holding means such as a ROM.
According to an eighth aspect of the present invention, there is provided the stereo image display apparatus according to the sixth aspect, wherein the binocular parallax control means effectively controls a deviation to make zero, where the deviation is the result of comparison of the predetermined desired eyeball accommodation and the eyeball accommodation detected by the eyeball accommodation detecting means.
According to the eighth aspect of the present invention, in addition to the advantages obtainable according to the sixth aspect of the present invention, a feedback control or like control is made to make zero the difference between a desired eyeball accommodation and an eyeball accommodation detected by eyeball accommodation detecting means as a comparison result, and adequate binocular parallax control thus can be quickly obtained following circumstance changes.
According to a ninth aspect of the present invention, there is provided the stereo image display apparatus according to the sixth aspect, wherein the desired vergence angle setting means includes data holding means which holds, as data representing the desired eyeball vergence angle, the value of an eyeball vergence angle obtained in connection with the display of left eye and right eye images having parallax corresponding to the distance from a virtual image position to the eyeball.
According to the ninth aspect of the present invention, in addition to the advantages obtainable according to the sixth aspect of the present invention, it is possible to realize an apparatus, which permits image viewing with less departure from the viewer""s natural sense of viewing and less fatigue of the eyes and serves of the viewer.
According to a tenth aspect of the present invention, there is provided the stereo image display apparatus according to the sixth aspect, wherein the desired vergence angle setting means which holds, as data representing the desired vergence angle, a measurement of the viewer""s eyeball vergence angle with respect to a reference stereo image set at a predetermined distance from the viewer.
According to the tenth aspect of the present invention, in addition to the advantages obtainable according to the sixth aspect of the present invention, it is possible to obtain binocular parallax control matched to the viewer""s actual eyeball vergence angle.
According to an eleventh aspect of the present invention, there is provided the stereo image display apparatus according to the sixth aspect, wherein the eyeball vergence angle detecting means includes a left eye image pick-up means for obtaining a left eye image, a right image pick-up means for obtaining a right eye image, and correlating means for obtaining a correlation between the left eye and right eye images.
According to the eleventh aspect of the present invention, in addition to the advantages obtainable according to the sixth aspect of the present invention, a specific method of realizing vergence angle detecting means is provided.
According to a twelfth aspect of the present invention, there is provided a stereo image display apparatus comprising: parallax distribution data generating means for generating parallax distribution data representing a parallax distribution status viewed in relation to a right eye (or left eye) image of left eye and right eye images having a binocular parallax at each predetermined position in the left eye (or right eye) display region; transmitting means for transmitting for transmitting left eye (or right eye) image data representing a left eye (or right eye) image and the parallax distribution data; receiving means for receiving the left eye (or right eye) image data and the parallax distribution data transmitted by the transmitting means; desired parallax data holding means for holding predetermined desired parallax data; parallax difference data deriving means for extracting data representing a parallax in particular coordinates among the parallax distribution data and deriving data representing the difference between the extracted parallax data and the desired parallax data held in the desired parallax data holding means; parallax distribution data correcting means for correcting the entire parallax distribution data by an equal amount according to the difference data from the parallax difference data deriving means; and image data generating means capable of generating right eye (or left eye) corrected image data representing the right eye (or left eye) image according to the left eye (or right eye) image data received by the receiving means and the corrected parallax distribution data from the parallax distribution data correcting means, and supplying the received left eye (or right eye) image data and the left eye (or right eye) corrected image data to predetermined display means.
According to the twelfth aspect of the present invention, the data amount concerning the transmission of stereo image data can be greatly reduced, and right eye (or left eye) corrected image data reconstructed in the receiving side system permits obtaining reconstructed images with controlled binocular parallax without having resort to separate horizontal display position shift means. It is thus possible to simplify the construction of the apparatus.
According to a thirteenth aspect of the present invention, there is provided a stereo image display apparatus comprising: parallax distribution data generating means for generating parallax distribution data representing a parallax distribution status viewed in relation to a right eye (or left eye) image of left eye and right eye images having a binocular parallax at each predetermined position in the left eye (or right eye) display region; transmitting means for transmitting for transmitting left eye (or right eye) image data representing a left eye (or right eye) image and the parallax distribution data; receiving means for receiving the left eye (or right eye) image data and the parallax distribution data transmitted by the transmitting means; desired parallax data holding means for holding predetermined desired parallax data; parallax difference data deriving means for extracting data representing a parallax in particular coordinates among the parallax distribution data and deriving data representing the difference between the extracted parallax data and the desired parallax data held in the desired parallax data holding means; parallax distribution data correcting means for correcting the entire parallax distribution data by an equal amount according to the difference data from the parallax difference data deriving means; and corrected image data generating means capable of processing the left eye (or right eye) image data received by the receiving means according to the corrected parallax distribution data from the parallax distribution data correcting means, thereby generating left eye (or right eye) corrected image data corresponding to a result of motion of the left eye (or right eye) image in a first sense of the horizontal direction and right eye (or left eye) corrected image data corresponding to a result of motion of the right eye (or left eye) corrected image data in a second sense of the horizontal direction opposite the first sense, such that the horizontal position of a stereo image perceived from a left eye and a right eye image formed by the left eye and right eye corrected image data is effectively equivalent to the horizontal position of a stereo image perceived by the left eye and right eye images, and supplying the generated left eye and right eye corrected image data to predetermined display means.
According to the thirteenth aspect of the present invention, the data amount concerning the transmission of stereo image data can be greatly reduced, and the horizontal display position of stereo image provided by right eye and left eye corrected image data reconstructed in the receiving side system, is not undesirably shifted.
According to a fourteenth aspect of the present invention, there is provided a stereo image display apparatus comprising: receiving means for receiving parallax distribution data representing a parallax distribution status viewed in relation to a right eye (or left eye) image of left eye and right eye images having a binocular parallax at each predetermined position in the left eye (or right eye) display region and left eye (or right eye) image data representing a left eye (or right eye) image; desired parallax data holding means for holding predetermined desired parallax data; parallax difference data deriving means for extracting data representing a parallax in particular coordinates among the parallax distribution data and deriving data representing the difference between the extracted parallax data and the desired parallax data held in the desired parallax data holding means; parallax distribution data correcting means for correcting the entire parallax distribution data by an equal amount according to the difference data from the parallax difference data deriving means; and image data generating means capable of generating right eye (or left eye) corrected image data representing the right eye (or left eye) corrected image corresponding to the right eye (or left eye) image on the basis of the left eye (or right eye) image data received by the receiving means and the corrected parallax distribution data from the parallax distribution data correcting means, and supplying the received right eye (or left eye) image data and the left eye (or right eye) corrected image data to predetermined display means.
According to the fourteenth aspect of the present invention, the data amount concerning the transmission of stereo image data can be greatly reduced, and the horizontal display position of stereo image provided by right eye (or left eye) corrected image data reconstructed in the receiving side system permits obtaining reconstructed images with controlled binocular parallax shift means. It is thus possible to simplify the construction of the apparatus.
According to a fifteenth aspect of the present invention, there is provided a stereo image display apparatus comprising: receiving means for receiving parallax distribution data representing a parallax distribution status viewed in relation to a right eye (or left eye) image of left eye and right eye images having a binocular parallax at each predetermined position in the left eye (or right eye) display region and left eye (or right eye) image data representing a left eye (or right eye) image; desired parallax data holding means for holding predetermined desired parallax data; parallax difference data deriving means for extracting data representing a parallax in particular coordinates among the parallax distribution data and deriving data representing the difference between the extracted parallax data and the desired parallax data held in the desired parallax data holding means; parallax distribution data correcting means for correcting the entire parallax distribution data by an equal amount according to the difference data from the parallax difference data deriving means; and corrected image data generating means capable of processing the left eye (or right eye) image data received by the receiving means according to the corrected parallax distribution data from the parallax distribution data correcting means, thereby generating left eye (or right eye) corrected image data corresponding to a result of motion of the left eye (or right eye) image in a first sense of the horizontal direction and right eye (or left eye) corrected image data corresponding to a result of motion of the right eye (or left eye) corrected image data in a second sense of the horizontal direction opposite the first sense, such that the horizontal position of a stereo image perceived from a left eye and a right eye image formed by the left eye and right eye corrected image data is effectively equivalent to the horizontal position of a stereo image perceived by the left eye and right eye images, and supplying the generated left eye and right eye corrected image data to predetermined display means.
According to the fifteenth aspect of the present invention, the data amount concerning the transmission of stereo image data can be greatly reduced, and the horizontal display position of stereo image provided by right eye and left eye corrected image data reconstructed in the receiving side system, is not undesirably shifted.
According to a sixteenth aspect of the present invention, there is provided the stereo image display apparatus according to one of the twelfth to fifteenth aspects, wherein the parallax difference data deriving means extracts data representing a parallax at coordinates as a point viewed by the viewer and obtains difference data between the extracted data and the desired parallax data held by the desired parallax data holding means.
According to the sixteenth aspect of the present invention, in addition to the advantages obtainable according to the twelfth to fifteenth aspects of the present invention, a stereo image control can be obtained such as to obtain instantaneous binocular parallax matched to the viewer""s viewing point.
According to a seventeenth aspect of the present invention, there is provided the stereo image display apparatus according to the aspect of the twelfth to fifteenth aspects, which further comprises data compressing means for compressing the left eye (or right eye) image and the parallax distribution data before transmission, and data expanding means for expanding the compressed left eye (or right eye) image and parallax distribution data after transmission.
According to the seventeenth aspect of the present invention, in addition to the advantages obtainable according to the twelfth to fifteenth aspects of the present invention, the amount of transmission data can be sufficiently compressed, permitting channel occupation time reduction and transmission efficiency increase.
Other objects and features will be clarified from the following description with reference to attached drawings.